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ABSTRACT 

The analytical process used in building a compendium 
of examples of accomplishments by the National Water Quality 
Initiative (NWQI) of the Cooperative Extension System is presented 
and discussed. Lessons learned are cited, and recommendations for 
future efforts to synthesize evidence on major program thrusts are 
made. The NWQI is focused on protecting the quality of the nation's 
groundwater and surface water through encouraging voluntary actions 
by farmers, ranchers, householders, and the general public. It has 
been reported that approximately 650 full-time equivalents of 
professional extension resources have been allocated annually (1989 
through 1990) to the initiative. Statistical information in fiscal 
year 1989 and 1990 reports was analyzed and synthesized to compile 
summary tables for all states and detailed tables f',r each state. In 
addition to this quantitative data, qualitative data from narrative 
portions of the reports were content-analyzed and summarized. 
Limitations in the reporting system and in the water quality reports 
were identified. The following types of indicators were selected to 
be tracked in the future: (1) rates of pollution; (2) rates of use of 
potential contributors to pollution; (3) extension-promoted practices 
to reduce potential contributions to pollution; and (4) extension 
strategy and activities to induce or accelerate adoption of such 
practices. One figure illustrates program components, and one table 
lists the planned indicators. (SLD) 
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Evaluating M^jor Program Thrusts: 
A Conceptual Framework of Evaluation Considerations and 
A Case Study of Synthesizing Evidence 
From a National Reporting System 

Satish Verma 
Division Leader (Personnel and Program Support) 
Louisiana Cooperative Extension Service 

Claude F. Bennett 
Program Evaluation Leader 
Extension Service, U.S. Department of Agriculture 

Program Plausibility 

The construction of a plausible and sensible model of how a program is supposed to work is a prerequisite to 
sound programming. Bickham (1987) equates program modeling with program theory in evaluation. According 
to Wholey (1987), construction of the program model involves identification of resources, activities and outcomes 
of a program and the causal assumptions that connect these. Assessing a program to see if it follows a defined 
logic provides evaluators with evidence that a plausible program exists. 

As programs are planned and undertaken to achieve desired objectives, the sequence of events and the 
underlying activities which would lead to the accomplishment of these objectives often are not laid out as a 
specific design. Thus, even if the desired objectives are achieved, it is questionable whether the achievement 
resulted from the program. Ensuring that a plausible program exists lends confidence to a decision to evaluate 
the results of the program. 

Program plausibility and evaluability is a legitimate concern for evaluators. This concern is magnified in the case 
of programs which are broad-scoped and permit program deliverers and/or partners considerable flexibility in 
program planning, implementation, evaluation and reporting. 

Cooperative Extension System Programs 

A prime example of the concern for program plausibility and evaluability is programming in the nationwdde 
Cooperative Extension System (CES), a three-way partnership of federal, state and local entities, focused on 
needs and issues of society and based in the research-teaching complex of state and territorial land grant colleges. 
In comparison to programs of straight-line agencies, CES programs are diverse and generally not governed by 
specific national objectives and goals. This diversification is a major strength of extension education but presents 
significant obstacles to national evaluation/reporting where aggregation of data and similarity of objectives are 
being sought (Heckel, 1981). Particularly susceptible are major national and state program thrusts which specify 
broad parameters and allow a great deal of programmmg flexibility to account for state and local conditions, 
respectively. 
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As a specific example of lack of program plausibility, assessments of water quality program efforts in three states 
- Maryland (1989), Virginia (1990) and Louisiana (1990) - pointed up what Wholey (1987) identifies as two 
major problems inhibiting the use of program evaluation by key policy makers, managers and staff. These were 
(a) lack of definition of the problems to be addressed, the program intervention, the expected outcomes of the 
program, or the expected impact on the problems to be addressed, and (b) lack of a clear logic of testable 
assumptions linking expenditure of program resources, the implementation of the program, and the resulting 
impact. Having discovered that there was no plausible program in place, the staff in these states constructed 
program logic models, turning the evaluation exercise into a program design effort. 

At the federal level, the current connotation of the term major program thrust is a national initiative or, 
alternati^'ely, a base program. National initiatives are the CES's commitment to respond to important societal 
problems of broad national concern with additional resources and significantly increased effort to achieve a major 
impact. Base programs are the major educational efforts central to the mission of the CES and common to most 
Extension units (Extension Service, USDA and Extension Committee on Organization and Policy, 1990). 
National initiatives and base programs provide a programmatic framework and direction to state Extension 
Services as they try to integrate national needs and issues with state and local needs. Currently, seven national 
initiatives and six national base programs comprise the national extension program. Additionally, in response 
to specific needs, special funding is provided for programs. 

At the state level, major program thrusts usually include some or all of the federal program thrusts and may also 
cover unique and specific programs significant to states. 

Usually, major program thrusts in the CES are broad in scope, technically and programmatically complex, may 
involve several disciplines, potentially be planned and unplemented at a number of program sites across the 
country for the benefit of multiple audiences, and at these program sites, encompass a breadth and depth of 
program efforts tailored to site-specific needs. This great diversity of program environments and characteristics 
have, over the years, made it difficult for CES program leaders to evaluate and/or report program outcomes 
which would satisfy legislative and public accountability needs. The task is made easier when major program 
thrusts are centrally directed, coordinated and monitored, such that evaluation and/or reporting systems 
procedures can be standardized. This type of evaluation planning gives program coordinators/evaluators a 
degree of control over evaluation/reporting methodology, enables aggregation of quantitative outcome data from 
multiple program sites and lends to the observed results and conclusions drawn some measure of confidence and 
generalizability. Another viable option which provides control to program evaluators is national and/or state 
impact studies focusing on specific programs. 

Typically, CES programs, except for those that are specially funded, do not have a prescribed 
evaluation/reporting system to report program outcome. State Cooperative Extension Services have reporting 
formats established for counties to report to state programs, and the Extension Service-USDA (ES-USDA, the 
federal partner) suggests formats for states to follow in reporting on national programs. Over the last eight years 
(1983-91), ES-USDA prescribed to state Extension Services a reporting procedure which included (a) a narrative 
description of accomplishments and (b) a quantitative expression of impact or outcome using key indicators. 
States reported on programs significant to them. Therefore, a variety of programs were reported on and, even 
with programs common to a number of states, there was considerable diversity of key program indicators and 
measurement units. It was only in a few national impact studies of significant programs that ES could provide 
a description of the programs as they were conducted across the country and program impacts that occurred as 
a result of the effort. 

Synthesizing State Reports on CES's National Water Quality Initiative 

The purpose of this paper is twofold: (1) to present and discuss the analytical process used in building a 
compendium of examples of accomplishments by CES*s National Water Quality Initiative, as reported during FY 
1989-91, and (2) to cite lessons learned and make recommendations which could be useful in shaping future 
efforts to synthesize evidence on major program thrusts. 



CES's National Initiative in Water Quality is focused on protecting the quality of the nation*s ground water and 
surface water. Launched in 1988, the Initiative is an extensive, nationwide effort to encourage voluntary actions 
by farmers, ranchers, householders, and the general public. This extensive, continuing, broad-scopf^d, focused 
effort involves statewide water quality programming by every state/territorial Extension Service in the country. 
It has been reported that approximately 650 Full-time Equivalents of professional Extension resources have been 
allocated aimually to the initiative. 

Improved Program Support (IPS) funds to strengthen the capacity of State Extension Services to deliver water 
quality programs were provided annually, from FY 1989 through FY 1991. The initiative has five program 
components, namely Nutrient Management, Pesticide Management, Animal Waste Management, Drinking 
Water/Wellhead Protection and Public Policy Education. 

The goal of the Initiative is to prevent /reduce the impairment of water through judicious use of crop nutrients, 
pesticides, and animal wastes in agricultural operations; protection of drinking water supplies; and collective 
actions by citizens at the local level to sustain the quality of surface and groundwater resources. Collection of 
direct evidence of protection/improvement of water quality generally requires extensive resources and long time 
periods. Therefore, intermediate indicators are used in reporting examples of water quality program 
accomplishments. These indicators are represented in Figure 1. 

ES-USDA prescribed a format in July 1989 for state/territory Extension Services to use in providing annual 
reports on their water quality programs. The format was designed to obtain statistical and narrative information 
on program processes and accomplishments. Statistical information was requested on specific practices in the 
fine program components. These practices are shown in Table 1. Besides these quantitative impact measures, 
narrative accounts of program delivery were to be provided in aimual reports. 

Electronic copies of annual reports when received by ES-USDA were entered into an annual accomplishments 
data base. A search-retrieval program was used to retrieve and prmt hard copies of the reports for review and 
analysis. 

Quantitative data analysis . Statistical information in the FY 1989 and 90 reports was analyzed and synthesized 
according to the steps presented below. Information for 1991 was not included in the analysis because these data 
were unavailable at the time the analysis was done. 

1. Five data categories were established: nutrient/animal waste management, drinking water/ wellhead 
protection, staff training, educational methods and materials, and educational contacts. Pesticide management 
was omitted because virtually no information was found in the reports on the prescribed practices. 

2. For each data category, a set of quantitative data items was identified; the data items were either 
prescribed by ES-USDA or empirically developed. For example, nutrient /animal waste management had 10 data 
items correspondmg to the practices prescribed in the ES-USDA guidelines format. Some of these items and 
measurement units were: 

soil testing - number of tests, number of acres 

soil nitrate testing - number of tests, number of acres 

nutrient management plans - number of plans, number of acres 

Appropriate data items in the categories of staff training, educational methods and materials, and educational 
contacts were constructed as the reports were reviewed. For example, data items under educational methods 
and materials included on-farm demonstrations and tests; workshops, seminars, conferences; audiovisual materials 
(videotapes, slide tapes, exhibits); etc. Educational contacts data items included different types of audiences such 
as farmers, householders, youth, etc. 

3. A dBase file was created for the designated data items, numbers entered, and detailed reports retrieved 
for each state. 



4. Summary tables for all states and detailed tables for each state comparing 1989 and 1990 information 
by data categories and data items were compiled. 

Qualitative data analysi s. Narrative information retrieved from the 1989-91 reports was content analyzed to 
identify and describe examples of progress and accomplishments. This analysis was clustered around the 
quantitative data items in the several data categories, and the information entered into WordPerfect files. The 
information was summarized as needed, in order to report and higlilight significant findings. 

Limitations of the Reporting System 

There are inherent limitations in utilizing data from the non-directive reporting system that CES had during the 
period of analysis, 1989-1991. 

1. The validity, and consequently, the generalizability of quantitative data is affected by incomplete data 
and/or estimates. 

2. Quantitative data items (names and measurement units) in reports that do not correspond exactly with 
the labels given to data items established for summarization purposes are subject to the interpretations and 
judgments of individual data analysts. 

3. Content analysis of narrative qualitative data may be affected by data analyst bias for inclusion in 
summary reports as examples of educational impacts. 

Limitations of Water Quality Reports 

Limitations were found in the water quality reports submitted by states. These limitations led to the following 
conditions and/or assumptions which affected the compendium of examples of accomplishments: 

1. Reports from only 31 states in 1989, and from only 44 states and the District of Columbia in each of 
1990 and 1991 were found to be present in the respective electronic databases. 

2. The quantitative data on Extension program impacts requested by ES-USDA were missing for most 
pesticide management practices and for some nutrient management practices. 

3. If state Extension Services did not report relevant data in a category/item, it was assumed that one or 
more of the following conditions existed: (a) the state Extension Service did not conduct programs in the 
category/item; (b) the state did not have a viable system for collecting data for reporting in response to the 
ES-USDA guidelines; (c) the state did not gather the particular data sought; and/or (d) the state chose not to 
include the requested data in its reports to ES-USDA. 

4. Different measures were used to report numerical data. For example, an indicator like soil nitrate 
testing was variably reported as number of tests, number of producers, number of crop acres affected, or a 
combination thereof. Depending on the range of possibilities, aggregation across states reporting on a particular 
item was virtually impossible. It was also difficult to interpret this kind of disparate information. 

5. Ax times, only general statements were made and actual quantities of the impact measures were not 
reported. For example, a statement such as "354 producers in 35 counties reduced rates of nitrogen fertilizer 
application" omitted the extent of reduced fertilizer application and, therefore, could not be included in the 
analysis. 

In view of the above limitations, and learning from the experiences gained in synthesizing 1989-91 reports, 
ES-USDA's 1992-95 water quality planning and reporting system has been re-designed to acquire comparable 
data from all state Extension Services on predetermined impact indicators. When summarized, these may 
provide an indication of national program accomplishments. 



Plans for Reporting Accomplishments of the National Water Quality Initiative, FY 1992-1995 



The current 4- year Plan of Work of CES (FY 1992-1995) entails comprehensive, national or multi-state reporting 
of program plans and accomplishments of the National Water Quality Initiative. This is in contrast to the 
present state-by-stale, selective reporting for FY 1989-1991. During FY 1992-1995, state Extension Services will 
report program plans and accomplishments within defined geographic areas where, currently or potentially, there 
is an unacceptable extent of water contamination which is being addressed througli one or more of the 5 program 
components described above. 

In the defined geographic areas, the following types of indicators will be tracked in regard to the 5 national 
program objectives: (a) rates of pollution (e.g., of domestic-use wells), (b) rates of use of potential contributors 
to pollution (e.g., pesticide application rates), (c) Extension-promoted practices to reduce potential contributions 
(e.g., recommended crop rotations), and (d) Extension strategy and activities to induce or accelerate adoption 
of such practices. 

These four types of indicators have been chosen to represent and to encourage the development and 
implementation of a program logic for each of the five program components of the Water Quality Initiative. 

As far as possible, common variables will be tracked across similar water quality programs implemented by state 
Extension Services addressing similar types of water quality problems. Indicator data supplied by the state 
Extension Services to Extension Service-USDA should allow for substantial cross-state analyses on numerous 
variables of interest to national policy makers, legislators, and Extension administrative and program leaders. 

The intent of the new approach to reporting accomplishments described above is to more fully meet 
requirements for program accountability and strengthen program management of the national Extension Water 
Quality Initiative. 

Implications and Recommendations 

The ES Water Quality Team has made progress toward increasing the quality and useabilily of state/territory 
plans of work and accomplishment reports. Almost 80% of states/territories participated in multi-state 
audioconferences. A series of information sheets were distributed with positive results. Water Quality staff of 
the state Extension Services are found to be generally in agreement with and supportive of the national indicators 
selected. 

The ES-USDA review of FY 1993 Plan of Work Updates is well underway. Conference calls have been held 
with several individual states to clarify data needs toward finalizing their POW Updates for FY 1993. 

Despite these efforts, it is not yet apparent whether the new system for national planning and reporting will be 
successful. ES's new system cannot be successful unless state Extension Services are aware of its nature and 
basis for approval of federal funding of state Extension Service water quality programs. 

In order to be successful, the new systems must reverse the track record of FY 1989-1991, which had severe 
limitations as noted above. In each of the years FY 1989-1991, ES received no Annual Report from several of 
the state Extension Services. Reports from the territories were not received. 

Only about 100 statements of accomplishment could be gleaned, from the three yeari» of reporting, for inclusion 
in a published compendium of examples of accomplishments. Of these, only about 50 were sufficiently complete 

— in narrative and/or quantitative description of accomplishments, and Extension's role in bringing them about 

— to merit inclusion in a publication (for external distribution) of selected examples of Extension Water Quality 
Initiative accomplishments. These 50 selected examples came from about one-half the state Extension Services. 
About one-quarter of the state Extension Services supplied most of the useable examples of program 
accomplishments. (The publications will be available for distribution in January 1993.) 



About one-third of the state Extension Services have submitted their FY 1993 POW Updates. The RS Water 
Quality Team is expending a great deal of time on the process of reviewing submitted 1993 POWs. It has been 
found necessary to request one or more re-submissions from each state Extension Service prior to ES approval 
for federal funding. Indicator data of the POW Updates submitted thus far are often far from complete and often 
contain data of questionable face validity. This requires a considerable amount of negotiation and re- negotiation 
on the part of ES with the state Extension Service. Of course, there is a range of quality of reporting, with some 
state Extension Services' first submissions of POWs being close to adequate. Only a few states have included 
success stories and exemplary data in iheir submissions, 

The great deal of t^me and attention given by the ES staff to education of the State Water Quality Coordinators 
regarding the planning and reporting systems, review of submitted POWs, and requests and repeated requests 
for corrected submissions, cannot overcome or compensate for lack of sufficient attention to POW and AR 
preparation by some of the state /territorial Extension Services. 

ES should reinforce to State Extension directors, agriculture and natural resource leaders, and water quality 
coordinators that more needs to be done to operationalize the 1990 agreement between ECOP and ES-USDA. 
The agreement was that, in return for ES narrowing the focus of its requests, states would provide high quality 
reporting in response to the indicators that ES identifies. 

ES should issue a policy pronouncement to state Extension Services that would serve to strengthen the notion 
that, when it comes to national plan of work and accomplishment report submissions, the image should be one 
of the Cooperative Extension System, not a "commonwealth of independent states." Such a policy 
pronouncement will serve as the "other side of the equation" to an effective ES program leader role in working 
with state counter-parts to build a national reporting system that provides for national accountability and 
improved national program management. 
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Table 1. Indicators Prescribed by ES-USDA for Planning and Reporting Impact of National Water Quality 
Initiative, 1989-91 



Nutrient Management (number of farms, number of acres, percent of total crop acres, crops) 

0 Soil tests to determine fertilizer rates 

o Appropriate nutrient credits from animal wastes/crop residues 

o Storage and application of manure as a nutrient source 

0 Manure testing for nutrients and spreader calibration 

o Recommended rates of nitrogen fertilizer application 

o Timing fertilizer application 

o Banding or injecting fertilizer into crop rows 

o Use of legume-based crop rotations 

0 Use of irrigation scheduling 

o Net reductions/incredses in application of nitrogen by crops(s) (pounds per acre/total tons) 

Pesticide Management (number of farms, number of acres, percent of total crop acres, crops) 

o Use of scouting information to decide pesticide applications 

o Selecting pesticides on water quality risk 

o Selecting short half-life pesticides 

o Selecting low leachability pesticides 

o Pesticide applications on biological modeling 

o Selecting pest resistant crop varieties 

o Rotating crops to hinder pest establishment and survival 

o Tillage methods and/or planting dates to deny pest habitats 

o Timing pesticide applications to protect natural enemies of pests 

o Net reductions/increases in application of pesticides/active ingredients (pounds per acre/total tons) 

o Pesticide selections that minimize water quality risks 

Drinking Water /Wellhead Protection 

0 Number of wells tested for bacteria/nutrients/pesticides 

o Number of wells tested for the first time 

o Number of wells tested regularly on schedule 

o Number of domestic wells disinfected 

o Water treatment devices installed 

o New v/ells drilled/constructed and utilized 

o Wells closed due to contamination 

o Use of least toxic household products 

o Safe disposal of toxic household/farmstead wastes 

o Scheduled septage maintenance 

Public Policy Education 

o New toxic/hazardous waste collection systems organized 

o County groundwater vulnerability maps prepared 

o Groundwater management districts established 

o Comprehensive solid waste policies adopted 

o Solid waste management programs improved 

o Policies/programs adopted/implemented to improve private waste disposal systems 

o Policies/programs adopted /implemented to improve community water systems 

o Landfill sitings and designs improved 
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